Mobile robot and control method for controlling the same

ABSTRACT

A mobile robot includes a travel drive unit configured to move a main body, an image acquisition unit configured to acquire a plurality of images by continuously photographing surroundings of the main body, a storage configured to store the plurality of continuous images acquired by the image acquisition unit, a sensor unit having one or more sensors configured to sense an object during the movement of the main body, and a controller configured to, in response to the sensing of the object by the sensor unit, select an image acquired at a specific point in time earlier than an object sensing time of the sensor unit from among the plurality of continuous images based on a moving direction and a moving speed of the main body, and recognize an attribute of the object included in the selected image acquired at the specific point in time.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase entry under 35 U.S.C. § 371from PCT International Application No. PCT/KR2017/009260, filed Aug. 24,2017, which claims the benefit of priority of Korean Patent ApplicationsNos. 10-2016-0108384, 10-2016-0108385, and 10-2016-0108386, all filedAug. 25, 2016, the contents of which are incorporated herein byreference in their entireties.

TECHNICAL FIELD

The present invention relates to a mobile robot and a method forcontrolling the same, and, more particularly, to a mobile robot capableof performing object recognition and avoidance and a method forcontrolling the mobile robot.

BACKGROUND ART

Robots have been developed for industry and has taken charge of aportion of factory automation. As the robots have recently been used inmore diverse fields, medical robots, aerospace robots, etc. have beendeveloped, and domestic robots for ordinary households are also beingmade. Among those robots, a robot capable of traveling on its own isreferred to as a mobile robot.

A typical example of mobile robots used in households is a robotcleaner, which cleans a predetermined area by suctioning ambient dust orforeign substances while traveling in the predetermined area on its own.

A mobile robot is capable of moving on its own and thus has freedom tomove, and the mobile robot is provided with a plurality of sensors toavoid an object during traveling so that the mobile robot may travelwhile detouring around the object.

In general, an infrared sensor or an ultrasonic sensor is used to sensean object by the mobile robot. The infrared sensor determines presenceof an object and a distance to the object based on an amount of lightreflected by the object or a time taken to receive the reflected light,whereas the ultrasonic sensor emits an ultrasound wave having apredetermined period and, in response to ultrasonic waves reflected byan object, determines a distance to the object based on a differencebetween the time to emit the ultrasonic waves and the time for theultrasonic waves to return to the object.

Meanwhile, object recognition and avoidance significantly affects notjust driving performance but also cleaning performance of the mobilerobot, and thus, it is required to secure reliability of an objectrecognizing capability.

A related art (Korean Patent Registration No. 10-0669892) discloses atechnology that embodies highly reliable object recognition by combiningan infrared sensor and an ultrasonic sensor.

However, the related art (Korean Patent Registration No. 10-0669892) hasa problem in its incapability of determining an attribute of an object.

FIG. 1 is a diagram for explanation of a method for sensing and avoidingan object by a conventional mobile robot.

Referring to FIG. 1, a robot cleaner performs cleaning by suctioningdust and foreign substance while moving (S11).

The robot cleaner recognizes presence of an object in response tosensing, by an ultrasonic sensor, an ultrasonic signal reflected by anobject (S12), and determines whether the recognized object has a heightthat the robot cleaner is able to climb (S13).

If it is determined the recognized object has a height that the robotcleaner is not able to climb, the robot cleaner may move in a forwarddirection (S14), and, otherwise, rotate by 90 degrees (S15).

For example, if the object is a low door threshold, the robot cleanerrecognizes the door threshold, and, if a recognition result indicatesthat the door threshold is low enough to pass through, the robot cleanermoves over the door threshold.

However, if an object determined to have a height which the robotcleaner is able to climb is an electric wire, the robot cleaner may bepossibly confined and thus restricted by the electric wire when movingover the electric wire.

In addition, since a fan base has a height similar to or lower than aheight of a threshold, the robot cleaner may determine that the fan baseis an object that the robot cleaner is able to climb. In this case, whenattempting to climb the fan base, the robot cleaner may be restrictedwith wheels spinning.

In addition, in the case where, not an entire human body, but a part ofhuman hair is sensed, the robot cleaner may determine that the humanhair has a height that the robot cleaner is able to climb, and travelforward, but, in this case, the robot cleaner may suction the human hairwhich possibly cause an accident.

Thus, a method for identifying an attribute of a front object and changea moving pattern according to the attribute is required.

Meanwhile, interests for machine learning such as artificialintelligence and deep learning have grown significantly in recent years.

Machine learning is one of artificial intelligence fields, and refers totraining computer with data and instruct the computer to perform acertain task, such as prediction and classification, based on the data.

The representative definition of machine learning is proposed byProfessor Tom M. Mitchell, as follows: “if performance at a specifictask in T, as measured by P, improves with experience E, it can be saidthat the task T is learned from experience E.” That is, improvingperformance with respect to a certain task T with consistent experienceE is considered machine learning.

Conventional machine learning focuses on statistics-basedclassification, regression, and clustering models. In particular,regarding map learning of the classification and regression models, ahuman defines properties of training data and a learning model ofdistinguishing new data based on the properties.

Meanwhile, unlike the conventional machine learning, deep learningtechnologies have been being developed recently. Deep learning is anartificial intelligence technology that enables a computer to learn onits own like a human on the basis of Artificial Neural Networks (ANN).That is, deep learning means that a computer discovers and determinesproperties on its own.

One of factors having accelerated development of deep learning may be,for example, deep learning frameworks that is provided as an opensource. For example, deep learning frameworks include Theano developedby University of Montreal, Canada, Torch developed by New YorkUniversity, U.S.A., Caffe developed by University of California,Berkeley, TensorFlow developed by Google, etc.

With disclosure of deep learning frameworks, a learning process, alearning method, and extracting and selecting data used for learninghave become more important in addition to a deep learning algorithm forthe purposes of effective learning and recognition.

In addition, more and more efforts are being made to use artificialintelligence and machine learning in a diversity of products andservices.

DISCLOSURE Technical Problem

One object of the present invention is to provide a mobile robot and amethod for controlling the mobile robot, the mobile robot which iscapable of acquiring image data for improving accuracy in objectattribute recognition.

Another object of the present invention is to provide a mobile robot,which is capable of determining an attribute of an object and adjustinga traveling pattern according to the attribute of the object to therebyperform highly reliable object recognition and avoidance, and a methodfor controlling the mobile robot.

Yet another object of the present invention is to provide a mobile robotand a method for controlling the mobile robot, the mobile robot which iscapable of moving forward, moving backward, stopping, detouring, and thelike according to a recognition result on an object, thereby enhancingstability of the mobile robot and user convenience and improvingoperational efficiency and cleaning efficiency.

Yet another object of the present invention is to provide a mobile robotcapable of accurately recognizing an attribute of an object based onmachine learning, and a method for controlling the mobile robot.

Yet another object of the present invention is to provide a mobile robotcapable of performing machine learning efficiently and extracting datafor object attribute recognition, and a method for controlling themobile robot.

Technical Solution

In order to achieve the above or other objects, a mobile robot in onegeneral aspect of the present invention includes: a travel drive unitconfigured to move a main body; an image acquisition unit configured toacquire a plurality of images by continuously photographing surroundingsof the main body; a storage configured to store the plurality ofcontinuous images acquired by the image acquisition unit; a sensor unithaving one or more sensors configured to sense an object during themovement of the main body; and a controller is configured to, when thesensor unit senses the object, select, based on a moving direction and amoving speed of the main body, an image acquired at a specific point intime earlier than an object sensing time of the sensor unit from amongthe plurality of continuous images, and recognize an attribute of theobject included in the selected image acquired at the specific point intime, and accordingly, it is possible to recognize an attribute of anobstacle accurately and register and manage an object area.

In addition, in order to achieve the above or other objects, a mobilerobot in one general aspect of the present invention includes a travelcontrol module configured to control driving of a travel control unitbased on a final attribute of a recognized object, and accordingly, itis possible to improve stability, user convenience, operationefficiency, and cleaning efficiency.

In addition, in order to achieve the above or other objects, a methodfor controlling a mobile robot in one general aspect of the presentinvention may include: acquiring a plurality of images by continuouslyphotographing surroundings of a main body during traveling; storing theplurality of continuous images acquired by the image acquisition unit;sensing an object by the sensor unit; when the sensing unit senses theobject, select, based on a moving direction and a moving speed of themain body, an image acquired at a specific point in time earlier than anobject sensing time of the sensor unit from among the plurality ofcontinuous images; recognizing an attribute of the object included inthe selected image acquired at the specific point in time; andcontrolling driving of the travel control unit based on the recognizedattribute of the object.

Advantageous Effects

According to at least one of the embodiments of the present invention,it is possible to acquire image data that helps increase accuracy ofrecognizing an attribute of an object.

According to at least one of the embodiments of the present invention,since a mobile robot is able to determine an attribute of an objectaccurately and adjust a traveling pattern according to the attribute ofthe object, it is possible to perform highly reliable object recognitionand avoidance.

In addition, according to at least one of the embodiments of the presentinvention, it is possible to provide a mobile robot, which is capable ofmoving forward, moving backward, stopping, detouring, and the likeaccording to a recognition result on an object to enhance userconvenience and improve operational efficiency and cleaning efficiency,and a method for controlling the mobile robot.

In addition, according to at least one of the embodiments of the presentinvention, it is possible to provide a mobile robot and a method forcontrolling the mobile robot, the mobile robot which is capable ofaccurately recognizing an attribute of an object based on learningmachine.

In addition, according to at least one of the embodiments of the presentinvention, it is possible for a mobile robot to perform machine learningefficiently and extract data for recognition

Meanwhile, other effects may be explicitly or implicitly disclosed inthe following detailed description according to embodiments of thepresent invention.

DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for explanation of a method for sensing and avoidingobject by a conventional robot cleaner.

FIG. 2 is a perspective view of a mobile robot according to anembodiment of the present invention and a base station for charging themobile robot.

FIG. 3 is a view of the top section of the mobile robot in FIG. 2.

FIG. 4 is a view of the front section of the mobile robot in FIG. 2.

FIG. 5 is a view of the bottom section of the mobile robot in FIG. 2.

FIGS. 6 and 7 are block diagram of a control relationship between majorcomponents of a mobile robot according to an embodiment of the presentinvention.

FIG. 8 is an exemplary schematic internal block diagram of a serveraccording to an embodiment of the present invention.

FIGS. 9 to 12 are diagrams for explanation of deep learning.

FIGS. 13 and 14 are diagrams for explanation of object recognition ofthe object recognition module.

FIG. 15 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

FIG. 16 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

FIGS. 17 to 25 are diagrams for explanation of a method for controllinga mobile robot according to an embodiment of the present invention.

FIG. 26 is a diagram for explanation of an operation method of a mobilerobot and a server according to an embodiment of the present invention.

FIG. 27 is a flowchart of an operation method of a mobile robot and aserver according to an embodiment of the present invention.

FIG. 28 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

FIG. 29 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

FIGS. 30 to 34 are diagrams for explanation of a method for controllinga mobile robot according to an embodiment of the present invention

BEST MODE

Hereinafter, embodiments of the present invention will be described indetail with reference to the accompanying drawings. While the inventionwill be described in conjunction with exemplary embodiments, it will beunderstood that the present description is not intended to limit theinvention to the exemplary embodiments.

In the drawings, in order to clearly and briefly describe the invention,parts which are not related to the description will be omitted, and, inthe following description of the embodiments, the same or similarelements are denoted by the same reference numerals even though they aredepicted in different drawings.

In the following description, with respect to constituent elements usedin the following description, the suffixes “module” and “unit” are usedor combined with each other only in consideration of ease in thepreparation of the specification, and do not have or serve as differentmeanings. Accordingly, the suffixes “module” and “unit” may beinterchanged with each other.

A mobile robot 100 according to an embodiment of the present inventionrefers to a robot capable of moving on its own using wheels and thelike, and may be a domestic robot, a robot cleaner, etc. A robot cleanerhaving a cleaning function is herein after described as an example ofthe mobile robot, but the present invention is not limited thereto.

FIG. 2 is a perspective view of a mobile robot and a charging stationfor charging the mobile robot according to an embodiment of the presentinvention.

FIG. 3 is a view of the top section of the mobile robot illustrated inFIG. 2, FIG. 4 is a view of the front section of the mobile robotillustrated in FIG. 2, and FIG. 5 is a view of the bottom section of themobile robot illustrated in FIG. 2.

FIGS. 6 and 7 are block diagram of a control relationship between majorcomponents of a mobile robot according to an embodiment of the presentinvention.

Referring to FIGS. 2 to 7, a mobile robot 100, 100 a, or 100 b includesa main body 110, and an image acquisition unit 120, 120 a, and 120 b foracquiring an area around the main body 110.

Hereinafter, each part of the main body 110 is defined as below: aportion facing a ceiling of a cleaning area is defined as the topsection (see FIG. 3), a portion facing a floor of the cleaning area isdefined as the bottom section (see FIG. 5), and a portion facing adirection of travel in portions constituting a circumference of the mainbody 110 between the top section and the bottom section is defined asthe front section (see FIG. 4).

The mobile robot 100, 100 a, or 100 b includes a travel drive unit 160for moving the main body 110. The travel drive unit 160 includes atleast one drive wheel 136 for moving the main body 110. The travel driveunit 160 includes a driving motor (not shown) connected to the drivewheel 136 to rotate the drive wheel. Drive wheels 136 may be provided onthe left side and the right side of the main body 110, respectively, andsuch drive wheels 136 are hereinafter referred to as a left wheel 136(L)and a right wheel 136(R), respectively.

The left wheel 136(L) and the right wheel 136(R) may be driven by onedriving motor, but, if necessary, a left wheel drive motor to drive theleft wheel 136(L) and a right wheel drive motor to drive the right wheel136(R) may be provided. The travel direction of the main body 110 may bechanged to the left or to the right by making the left wheel 136(L) andthe right wheel 136(R) have different rates of rotation.

A suction port 110 h to suction air may be formed on the bottom surfaceof the body 110, and the body 110 may be provided with a suction device(not shown) to provide suction force to cause air to be suctionedthrough the suction port 110 h, and a dust container (not shown) tocollect dust suctioned together with air through the suction port 110 h.

The body 110 may include a case 111 defining a space to accommodatevarious components constituting the mobile robot 100, 110 a, 110 b. Anopening allowing insertion and retrieval of the dust containertherethrough may be formed on the case 111, and a dust container cover112 to open and close the opening may be provided rotatably to the case111.

There may be provided a roll-type main brush having bristles exposedthrough the suction port 110 h and an auxiliary brush 135 positioned inthe front of the bottom surface of the body 110 and having bristlesforming a plurality of radially extending blades. Dust is removed fromthe floor in a cleaning area by rotation of the brushes 134 and 135, andsuch dust separated from the floor in this way is suctioned through thesuction port 110 h and collected in the dust container.

A battery 138 serves to supply power necessary not only for the drivemotor but also for overall operations of the mobile robot 100, 100 a, or100 b. When the battery 138 of the robot cleaner 100 is running out, themobile robot 100, 100 a, or 100 b may perform return travel to thecharging base 200 to charge the battery, and during the return travel,the robot cleaner 100 may autonomously detect the position of thecharging base 200.

The charging base 200 may include a signal transmitting unit (not shown)to transmit a predetermined return signal. The return signal mayinclude, but is not limited to, a ultrasonic signal or an infraredsignal.

The mobile robot 100, 100 a, or 100 b may include a signal sensing unit(not shown) to receive the return signal. The charging base 200 maytransmit an infrared signal through the signal transmitting unit, andthe signal sensing unit may include an infrared sensor to sense theinfrared signal. The mobile robot 100, 100 a, or 100 b moves to theposition of the charging base 200 according to the infrared signaltransmitted from the charging base 200 and docks with the charging base200. By docking, charging of the robot cleaner 100 is performed betweena charging terminal 133 of the mobile robot 100, 100 a, or 100 b and acharging terminal 210 of the charging base 200.

In some implementations, the mobile robot 100, 100 a, or 100 b mayperform return travel to the base station 200 based on an image or in alaser-pattern extraction method.

The mobile robot 100, 100 a, or 100 b may return to the base station byrecognizing a predetermined pattern formed at the base station 200 usingan optical signal emitted from the main body 110 and extracting thepredetermined pattern.

For example, the mobile robot 100, 100 a, or 100 b according to anembodiment of the present invention may include an optical patternsensor (not shown).

The optical pattern sensor may be provided in the main body 110, emit anoptical pattern to an active area where the mobile robot 100, 100 a, or100 b moves, and acquire an input image by photographing an area intowhich the optical pattern is emitted. For example, the optical patternmay be light having a predetermined pattern, such as a cross pattern.

The optical pattern sensor may include a pattern emission unit to emitthe optical pattern, and a pattern image acquisition unit to photographan area into which the optical pattern is emitted.

The pattern emission unit may include a light source, and an OpticalPattern Projection Element (OPPE). The optical pattern is generated aslight incident from the light source passes through the OPPE. The lightsource may be a Laser Diode (LD), a Light Emitting Diode (LED), etc.

The pattern emission unit may emit light in a direction forward of themain body, and the pattern image acquisition unit acquires an inputimage by photographing an area where an optical pattern is emitted. Thepattern image acquisition unit may include a camera, and the camera maybe a structured light camera.

Meanwhile, the base station 200 may include two or more locator beaconsspaced a predetermined distance apart from each other.

A locator beacon forms a mark distinguishable from the surroundings whenan optical pattern is incident on a surface of the locator beacon. Thismark may be caused by transformation of the optical pattern due to amorphological characteristic of the locator beacon, or may be caused bydifference in light reflectivity (or absorptivity) due to a materialcharacteristic of the locator beacon.

The locator beacon may include an edge that forms the mark. An opticalpattern incident on the surface of the locator beacon is bent on theedge with an angle, and the bending is found as a cusp in an input imageas the mark.

The mobile robot 100, 100 a, or 100 b may automatically search for thebase station when a remaining battery capacity is insufficient, or maysearch for the base station even when a charging command is receivedfrom a user.

When the mobile robot 100, 100 a, or 100 b searches for the basestation, a pattern extraction unit extracts cusps from an input imageand a controller 140 acquire location information of the extractedcusps. The location information may include locations in athree-dimensional space, which takes into consideration of distancesfrom the mobile robot 100, 100 a, or 100 b to the cusps.

The controller 140 calculates an actual distance between the cusps basedon the acquired location information of the cusps and compare the actualdistance with a preset reference threshold, and, if a distance betweenthe actual distance and the reference threshold falls into apredetermined range, it may be determined that the charging station 200is found.

Alternatively, the mobile robot 100, 100 a, or 100 b may return to thebase station 200 by acquiring images of the surroundings with a cameraof the image acquisition unit 120 and extracting and identifying a shapecorresponding to the charging station 200 from the acquired image.

In addition, the mobile robot 100, 100 a, or 100 b may return to thebase station by acquiring images of surroundings with the camera of theimage acquisition unit 120 and identifying a predetermined opticalsignal emitted from the base station 200.

The image acquisition unit 120, which is configured to photograph thecleaning area, may include a digital camera. The digital camera mayinclude at least one optical lens, an image sensor (e.g., a CMOS imagesensor) including a plurality of photodiodes (e.g., pixels) on which animage is created by light transmitted through the optical lens, and adigital signal processor (DSP) to construct an image based on signalsoutput from the photodiodes. The DSP may produce not only a still image,but also a video consisting of frames constituting still images.

Preferably, the image acquisition unit 120 includes a front camera 120 aprovided to acquire an image of a front field of view from the main body110, and an upper camera 120 b provided on the top section of the mainbody 110 to acquire an image of a ceiling in a travel area, but theposition and capture range of the image acquisition unit 120 are notnecessarily limited thereto.

In this embodiment, a camera may be installed at a portion (ex. thefront, the rear, and the bottom) of the mobile robot and continuouslyacquire captured images in a cleaning operation. Such a camera may beinstalled in plural at each portion for photography efficiency. An imagecaptured by the camera may be used to recognize a type of a substanceexisting in a corresponding space, such as dust, a human hair, andfloor, to determine whether cleaning has been performed, or to confirm apoint in time to clean.

The front camera 120 a may capture an object or a condition of acleaning area ahead in a direction of travel of the mobile robot 100,100 a, or 100 b.

According to an embodiment of the present invention, the imageacquisition unit 120 may acquire a plurality of images by continuouslyphotographing the surroundings of the main body 110, and the pluralityof acquired images may be stored in a storage 150.

The mobile robot 100, 100 a, or 100 b may increase accuracy in objectrecognition by using a plurality of images or may increase accuracy ofrecognition of an object by selecting one or more images from aplurality of images to use effective data.

In addition, the mobile robot 100, 100 a, or 100 b may include a sensorunit 170 that includes sensors serving to sense a variety of datarelated to operation and states of the mobile robot.

For example, the sensor unit 170 may include an object detection sensorto sense an object present ahead. In addition, the sensor unit 170 mayfurther include a cliff detection sensor 132 to sense presence of acliff on the floor of a travel region, and a lower camera sensor 139 toacquire an image of the floor.

Referring to FIGS. 2 and 4, the object detection sensor 131 may includea plurality of sensors installed at a predetermined interval on an outercircumferential surface of the mobile robot 100.

For example, the sensor unit 170 may include a first sensor disposed onthe front surface of the main body 110, and second and third sensorsspaced apart from the first sensor in the left and right direction.

The object detection sensor 131 may include an infrared sensor, anultrasonic sensor, an RF sensor, a magnetic sensor, a position sensitivedevice (PSD) sensor, etc.

Meanwhile, the positions and types of sensors included in the objectdetection sensor 131 may differ depending on a model of a mobile robot,and the object detection sensor 131 may further include various sensors.

The object detection sensor 131 is a sensor for sensing a distance to awall or an object in an indoor space, and is, but not limited thereto,described as an ultrasonic sensor.

The object detection sensor 131 senses an object, especially an object,present in a direction of travel (movement) of the mobile robot andtransmits object information to the controller 140. That is, the objectdetection sensor 131 may sense a protrusion, home stuff, furniture, awall surface, a wall corner, etc. that are present in a path throughwhich the robot cleaner moves, and deliver related information to acontrol unit.

In this case, the controller 140 may sense a position of the objectbased on at least two signals received through the ultrasonic sensor,and control movement of the mobile robot 100 according to the sensedposition of the object.

In some implementations, the object detection sensor 131 provided on anouter side surface of the case 110 may include a transmitter and areceiver.

For example, the ultrasonic sensor may be provided such that at leastone transmitter and at least two receivers cross one another.Accordingly, it is possible to transmit a signal in various directionsand receive a signal, reflected by an object, from various directions.

In some implementations, a signal received from the object detectionsensor 131 may go through signal processing, such as amplification andfiltering, and thereafter a distance and a direction to an object may becalculated.

Meanwhile, the sensor unit 170 may further include an operationdetection sensor that serves to sense operation of the mobile robot 100,100 a, or 100 b upon driving of the main body 110, and output operationinformation. As the operation detection sensor, a gyro sensor, a wheelsensor, an acceleration sensor, etc. may be used.

When the mobile robot 100, 100 a, or 100 b moves according to anoperation mode, the gyro sensor senses a direction of rotation anddetects an angle of rotation. The mobile robot 100, 100 a, or 100 bdetects a velocity of the mobile robot 100, 100 a, or 100 b, and outputsa voltage value proportional to the velocity. The controller 140 maycalculate the direction of rotation and the angle of rotation using thevoltage value that is output from the gyro sensor.

The wheel sensor is connected to the left wheel 136(L) and the rightwheel 136(R) to sense the revolution per minute (RPM) of the wheels. Inthis example, the wheel sensor may be a rotary encoder. The rotaryencoder senses and outputs the RPM of the left wheel 136(L) and theright wheel 136(R).

The controller 140 may calculate the speed of rotation of the left andright wheels using the RPM. In addition, the controller 140 maycalculating an angle of rotation using a difference in the RPM betweenthe left wheel 136(L) and the right wheel 136(R).

The acceleration sensor senses a change in speed of the mobile robot100, 100 a, or 100 b, e.g., a change in speed according to a start, astop, a direction change, collision with an object, etc. Theacceleration sensor may be attached to a position adjacent to a mainwheel or a secondary wheel to detect the slipping or idling of thewheel.

In addition, the acceleration sensor may be embedded in the controller140 and detect a change in speed of the mobile robot 100, 100 a, or 100b. That is, the acceleration sensor detects impulse according to achange in speed and outputs a voltage value corresponding thereto. Thus,the acceleration sensor may perform the function of an electronicbumper.

The controller 140 may calculate a change in the positions of the mobilerobot 100, 100 a, or 100 b based on operation information output fromthe operation detection sensor. Such a position is a relative positioncorresponding to an absolute position that is based on imageinformation. By recognizing such a relative position, the mobile robotmay enhance position recognition performance using image information andobject information.

Meanwhile, the mobile robot 100, 100 a, or 100 b may include a powersupply (not shown) having a rechargeable battery 138 to provide power tothe robot cleaner.

The power supply may supply driving power and operation power to eachelement of the mobile robot 100, 100 a, or 100 b, and may be charged byreceiving a charging current from the charging station 200 when aremaining power capacity is insufficient.

The mobile robot 100, 100 a, or 100 b may further include a batterysensing unit to sense the state of charge of the battery 138 andtransmit a sensing result to the controller 140. The battery 138 isconnected to the battery sensing unit such that a remaining batterycapacity state and the state of charge of the battery is transmitted tothe controller 140. The remaining battery capacity may be disposed on ascreen of an output unit (not shown).

In addition, the mobile robot 100, 100 a, or 100 b includes an operator137 to input an on/off command or any other various commands. Throughthe operator 137, various control commands necessary for overalloperations of the mobile robot 100 may be received. In addition, themobile robot 100, 100 a, or 100 b may include an output unit (not shown)to display reservation information, a battery state, an operation mode,an operation state, an error state, etc.

Referring to FIGS. 6 and 7, the mobile robot 100 a, or 100 b includesthe controller 140 for processing and determining a variety ofinformation, such as recognizing the current position, and the storage150 for storing a variety of data. In addition, the mobile robot 100,100 a, or 100 b may further include a communication unit 190 fortransmitting and receiving data with an external terminal.

The external terminal may be provided with an application forcontrolling the mobile robot 100 a and 100 b, displays a map of a travelarea to be cleaned by executing the application, and designate aspecific area on a map to be cleaned. Examples of the external terminalmay include a remote control, a PDA, a laptop, a smartphone, a tablet,and the like in which an application for configuring a map is installed.

By communicating with the mobile robot 100 a and 100 b, the externalterminal may display the current position of the mobile robot inaddition to the map, and display information on a plurality of areas. Inaddition, the external terminal may update the current position inaccordance with traveling of the mobile robot and displays the updatedcurrent position.

The controller 140 controls the image acquisition unit 120, the operator137, and the travel drive unit 160 of the mobile robot 100 a and 100 bso as to control overall operations of the mobile robot 100.

The storage 150 serves to record various kinds of information necessaryfor control of the robot cleaner 100 and may include a volatile ornon-volatile recording medium. The recording medium serves to store datawhich is readable by a micro processor and may include a hard disk drive(HDD), a solid state drive (SSD), a silicon disk drive (SDD), a ROM, aRAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage,etc.

Meanwhile, a map of the travel area may be stored in the storage 150.The map may be input by an external terminal or a server capable ofexchanging information with the mobile robot 100 a and 100 b throughwired or wireless communication, or may be constructed by the mobilerobot 100 a and 100 b through self-learning.

On the map, positions of rooms within the cleaning area may be marked.In addition, the current position of the mobile robot 100 a and 100 bmay be marked on the map, and the current position of the mobile robot100 a and 100 b on the map may be updated during travel of the mobilerobot 100 a and 100 b. The external terminal stores a map identical to amap stored in the storage 150.

The storage 150 may store cleaning history information. The cleaninghistory information may be generated each time cleaning is performed.

The map of a travel area stored in the storage 150 may be a navigationmap used for travel during cleaning, a simultaneous localization andmapping (SLAM) map used for position recognition, a learning map usedfor learning cleaning by storing corresponding information uponcollision with an object or the like, a global localization map used forglobal localization recognition, an object recognition map whereinformation on a recognized object is recorded, etc.

Meanwhile, maps may be stored and managed in the storage 150 by purposesor may be classified exactly by purposes. For example, a plurality ofinformation items may be stored in one map so as to be used for at leasttwo purposes. For an instance, information on a recognized object may berecorded on a learning map so that the learning map may replace anobject recognition map, and a global localization position is recognizedusing the SLAM map used for position recognition so that the SLAM mapmay replace or be used together with the global localization map.

The controller 140 may include a travel control module 141, a positionrecognition module 142, a map generation module 143, and an objectrecognition module 144.

Referring to FIGS. 3 to 7, the travel control module 141 serves tocontrol travelling of the mobile robot 100, 100 a, or 100 b, andcontrols driving of the travel drive unit 160 according to a travelsetting. In addition, the travel control module 141 may determine atravel path of the mobile robot 100, 100 a, or 100 b based on operationof the travel drive unit 160. For example, the travel control module 141may determine the current or previous moving speed or distance travelledof the mobile robot 100 based on an RPM of the drive wheel 136, and mayalso determine the current or previous direction shifting process basedon directions of rotation of each drive wheel 136(L) or 136(R). Based ontravel information of the mobile robot 100, 100 a, or 100 b determinedin this manner, the position of the mobile robot 100, 100 a, or 100 b onthe map may be updated.

The map generation module 143 may generate a map of a travel area. Themap generation module 143 may write a map by processing an imageacquired by the image acquisition unit 120. That is, a cleaning mapcorresponding to a cleaning area may be written.

In addition, the map generation module 143 may enable globallocalization in association with a map by processing an image acquiredby the image acquisition unit 120 at each position.

The position recognition module 142 may estimate and recognize thecurrent position. The position recognition module 142 identifies theposition in association with the map generation module 143 using imageinformation of the image acquisition unit 120, thereby enabled toestimate and recognize the current position even when the position ofthe mobile robot 100, 100 a, or 100 b suddenly changes.

The mobile robot 100, 100 a, or 100 b may recognize the position usingthe position recognition module 142 during continuous travel, and theposition recognition module 142 may learn a map and estimate the currentposition by using the map generation module 143 and the objectrecognition module 144 without the position recognition module 142.

While the mobile robot 100, 100 a, or 100 b travels, the imageacquisition unit 120 acquires images of the surroundings of the mobilerobot 100. Hereinafter, an image acquired by the image acquisition unit120 is defined as an “acquisition image”.

An acquisition image includes various features located at a ceiling,such as lighting devices, edges, corners, blobs, ridges, etc.

The map generation module 143 detects features from each acquisitionimage. In computer vision, there are a variety of well-known techniquesfor detecting features from an image. A variety of feature detectorssuitable for such feature detection are well known. For example, thereare Canny, Sobel, Harris&Stephens/Plessey, SUSAN, Shi&Tomasi, Levelcurve curvature, FAST, Laplacian of Gaussian, Difference of Gaussians,Determinant of Hessian, MSER, PCBR, and Grey-level blobs detectors.

The map generation module 143 calculates a descriptor based on eachfeature. For feature detection, the map generation module 143 mayconvert a feature into a descriptor using Scale Invariant FeatureTransform (SIFT).

The descriptor is defined as a group of individual features present in aspecific space and may be expressed as an n-dimensional vector. Forexample, various features at a ceiling, such as edges, corners, blobs,ridges, etc. may be calculated into respective descriptors and stored inthe storage 150.

Based on descriptor information acquired from an image of each position,at least one descriptor is classified by a predetermined subordinateclassification rule into a plurality of groups, and descriptors includedin the same group may be converted by a predetermined subordinaterepresentation rule into subordinate representative descriptors. Thatis, a standardization process may be performed by designating arepresentative value for descriptors obtained from an individual image.

The SIFT enables detecting a feature invariant to a scale, rotation,change in brightness of a subject, and thus, it is possible to detect aninvariant (e.g., rotation-invariant) feature of an area even when imagesof the area is captured by changing a position of the mobile robot 100.However, aspects of the present invention are not limited thereto, andVarious other techniques (e.g., Histogram of Oriented Gradient (HOG),Haar feature, Fems, Local Binary Pattern (LBP), and Modified CensusTransform (MCT)) can be applied.

The map generation module 143 may classify at least one descriptor ofeach acquisition image into a plurality groups based on descriptorinformation, obtained from an acquisition image of each position, by apredetermined subordinate classification rule, and may convertdescriptors included in the same group into subordinate representativedescriptors by a predetermined subordinate representation rule.

In another example, all descriptors obtained from acquisition images ofa predetermined area, such as a room, may be classified into a pluralityof groups by a predetermined subordinate classification rule, anddescriptors included in the same group may be converted into subordinaterepresentative descriptors by the predetermined subordinaterepresentation rule.

Through the above procedure, the map generation module 143 may obtain afeature distribution for each position. The feature distribution of eachposition may be represented by a histogram or an n-dimensional vector.In another example, without using the predetermined subordinateclassification rule and the predetermined subordinate representationrule, the map generation module 143 may estimate an unknown currentposition based on a descriptor calculated from each feature.

In addition, when the current position of the mobile robot 100, 100 a,or 100 b has become unknown due to position hopping, the map generationmodule may estimate the current position of the mobile robot 100 basedon data such as a pre-stored descriptor and a subordinate representativedescriptor.

The mobile robot 100, 100 a, or 100 b acquires an acquisition unit imageat the unknown current position using the image acquisition unit 120.Various features located at a ceiling, such as lighting devices, edges,comers, blobs, and ridges, are found in the acquisition image.

The position recognition module 142 detects features from an acquisitionimage. Descriptions about various methods for detecting features from animage in computer vision and various feature detectors suitable forfeature detection are the same as described above.

The position recognition module 142 calculates a recognition descriptorbased on each recognition feature through a recognition descriptorcalculating process. In this case, the recognition feature and therecognition descriptor are used to describe a process performed by theobject recognition module 144, and to be distinguished from the termsdescribing the process performed by the map generation module 143.However, these are merely terms defined to describe characteristics of aworld outside the mobile robot 100, 100 a, or 100 b.

For such feature detection, the position recognition module 142 mayconvert a recognition feature into a recognition descriptor using theSIFT technique. The recognition descriptor may be represented by ann-dimensional vector.

The SIFT is an image recognition technique by which easilydistinguishable features, such as corners, are selected in anacquisition image and n-dimensional vectors are obtained, which arevalues of dimensions indicative of a degree of radicalness of change ineach direction with respect to a distribution feature (a direction ofbrightness change and a degree of radicalness of the change) of gradientof pixels included in a predetermined area around each of the features.

Based on information on at least one recognition descriptor obtainedfrom an acquisition image acquired at an unknown current position, theposition recognition module 142 performs conversion by a predeterminedsubordinate conversion rule into information (a subordinate recognitionfeature distribution), which is comparable with position information(e.g., a subordinate feature distribution for each position) subject tocomparison.

By the predetermined subordinate comparison rule, a feature distributionfor each position may be compared with a corresponding recognitionfeature distribution to calculate a similarity therebetween. Asimilarity (probability) for the position corresponding to eachposition, and a position having the highest probability may bedetermined to be the current position.

As such, the controller 140 may generate a map of a travel areaconsisting of a plurality of areas, or recognize the current position ofthe main body 110 based on a pre-stored map.

When the map is generated, the controller 140 transmits the map to anexternal terminal through the communication unit 190. In addition, asdescribed above, when a map is received from the external terminal, thecontroller 140 may store the map in the storage.

In addition, when the map is updated during traveling, the controller140 may transmit update information to the external terminal so that themap stored in the external terminal becomes identical to the map storedin the mobile robot 100. As the identical map is stored in the externalterminal and the mobile robot 100, the mobile robot 100 may clean adesignated area in response to a cleaning command from a mobileterminal, and the current position of the mobile robot may be displayedin the external terminal.

In this case, the map divides a cleaning area into a plurality of areas,includes a connection channel connecting the plurality of areas, andincludes information on an object in an area.

When a cleaning command is received, the controller 140 determineswhether a position on the map and the current position of the mobilerobot match each other. The cleaning command may be received from aremote controller, an operator, or the external terminal.

When the current position does not match with a position marked on themap, or when it is not possible to confirm the current position, thecontroller 140 recognizes the current position, recovers the currentposition of the mobile robot 100 and then controls the travel drive unit160 to move to a designated area based on the current position.

When the current position does not match with a position marked on themap or when it is not possible to confirm the current position, theposition recognition module 142 analyzes an acquisition image receivedfrom the image acquisition unit 120 and estimate the current positionbased on the map. In addition, the object recognition module 144 or themap generation module 143 may recognize the current position asdescribed above.

After the current position of the mobile robot 100, 100 a, or 100 b isrecovered by recognizing the position, the travel control module 141 maycalculate a travel path from the current position to a designated areaand controls the travel drive unit 160 to move to the designated area.

When a cleaning pattern information is received from a server, thetravel control module 141 may divide the entire travel area into aplurality of areas based on the received cleaning pattern informationand set one or more areas as a designated area.

In addition, the travel control module 141 may calculate a travel pathbased on the received cleaning pattern information, and perform cleaningwhile traveling along the travel path.

When cleaning of a set designated area is completed, the controller 140may store a cleaning record in the storage 150.

In addition, through the communication unit 190, the controller 140 maytransmit an operation state or a cleaning state of the mobile robot 100at a predetermined cycle.

Accordingly, based on the received data, the external terminal displaysthe position of the mobile robot together with a map on a screen of anapplication in execution, and outputs information on a cleaning state.

The mobile robot 100, 100 a, or 100 b moves in one direction untildetecting an object or a wall, and, when the object recognition module144 recognizes an object, the mobile robot 100, 100 a, or 100 b maydetermine a travel pattern, such as forward travel and rotationaltravel, depending on an attribute of the recognized object.

For example, if the attribute of the recognized object implies an objectthat the mobile robot is able to climb, the mobile robot 100, 100 a, or100 b may keep traveling forward. Alternatively, if the attribute of therecognized object implies an object that the mobile robot is not able toclimb, the mobile robot 100, 100 a, or 100 b may travel in a zigzagpattern in a manner in which the mobile robot 100, 100 a, or 100 brotates, moves a predetermined distance, and then moves a distance in adirection opposite to the initial direction of travel until detectinganother object.

The mobile robot 100, 100 a, or 100 b may perform object recognition andavoidance based on machine learning.

The controller 140 may include the object recognition module 144 forrecognizing an object, which is pre-learned through machine learning, inan input image, and the travel control module 141 for controllingdriving of the travel drive unit 160 based on an attribute of therecognized object.

The mobile robot 100, 100 a, or 100 b may include the object recognitionmodule 144 that has learned an attribute of an object through machinelearning.

The machine learning refers to a technique which enables a computer tolearn without a logic instructed by a user so as to solves a problem onits own.

Deep learning is an artificial intelligence technology that trains acomputer to learn human thinking based on Artificial Neural Networks(ANN) for constructing artificial intelligence so that the computer iscapable of learning on its own without a user's instruction.

The ANN may be implemented in a software form or a hardware form, suchas a chip and the like.

The object recognition module 144 may include an ANN in a software formor a hardware form, in which attributes of objects is learned.

For example, the object recognition module 144 may include a Depp NeuralNetwork (DNN) trained through deep learning, such as ConvolutionalNeural Network (CNN), Recurrent Neural Network (RNN), Deep BeliefNetwork (DBN), etc.

Deep learning will be described in more detail with reference to FIGS. 9to 12.

The object recognition module 144 may determine an attribute of anobject in input image data based on weights between nodes included inthe DNN.

Meanwhile, when the sensor unit 170 senses an object while the mobilerobot 100, 100 a, or 100 b moves, the controller 140 may perform controlto select, based on a moving direction and a moving speed of the mainbody 110, an image acquired at a specific point in time earlier than anobject sensing time by the sensor unit 170 from among a plurality ofcontinuous images acquired by the image acquisition unit 120.

In the case where the image acquisition unit 120 acquires an image usingan object sensing time of the sensor unit 170 as a trigger signal, anobject may not be included in the acquired image or may be captured in avery small size because the mobile robot keeps moving.

Thus, in one embodiment of the present invention, it is possible toselect, based on a moving direction and a moving speed of the main body110, an image acquired at a specific point in time earlier than anobject sensing time of the sensor unit 170 from among a plurality ofcontinuous images acquired by the image acquisition unit 120 and use theselected as data for object recognition.

Meanwhile, the object recognition module 144 may recognize an attributeof an object included in the selected image acquired at thepredetermined point in time, based on data that is pre-learned throughmachine learning.

In addition, when the sensor unit 170 senses an object while the mobilerobot 100, 100 a, or 100 b moves, the controller 140 may perform controlto extract part of an image acquired by the image acquisition unit 120in correspondence to a direction in which the object sensed by thesensor unit 170 is present.

The image acquisition unit 120, especially the front camera 120 a, mayacquire an image within a predetermined range of angles in a movingdirection of the mobile robot 100, 100 a, or 100 b.

The controller 140 may distinguish an attribute of the object present inthe moving direction, not by using the whole image acquired by the imageacquisition unit, especially the front camera 120 a, but by using only apart of the image.

Referring to FIG. 6, the controller 140 may further include an imageprocessing module 145 for extracting a part of an image acquired by theimage acquisition unit 120 in correspondence to a direction in which anobject sensed by the sensor unit 170 is present.

Alternatively, referring to FIG. 7, the mobile robot 100 b may furtherinclude an additional image processing unit 125 for extracting a part ofan image acquired by the image acquisition unit 120 in correspondence toa direction in which an object sensed by the sensor unit 170 is present.

The mobile robot 100 a in FIG. 6 and the mobile robot 100 b in FIG. 7are identical to each other, except for the image processing module 145and the image processing unit 125.

Alternatively, in some implementations, the image acquisition unit 120may extract a part of the image on its own, unlike the examples of FIGS.6 and 7.

The object recognition module 144 having trained through machinelearning has a higher recognition rate if a learned object occupies alarger portion of input image data.

Accordingly, the present invention may extract a different part of animage acquired by the image acquisition unit 120 depending on adirection in which an object sensed by the sensor unit 170 such as anultrasonic sensor, and use the extracted part of the image as data forrecognition, thereby enhancing a recognition rate.

The object recognition module 144 may recognize an object in theextracted part of the image, based on data that is pre-learned throughmachine learning.

In addition, the travel control module 141 may control driving of thetravel drive unit 160 based on an attribute of the recognized object.

Meanwhile, when the object is sensed from a forward-right direction ofthe main body, the controller 140 may perform control to extract a rightlower area of the image acquired by the image acquisition unit; when theobject is sensed from a forward-left direction of the main body, thecontroller 140 may perform control to extract a left lower area of theimage acquired by the image acquisition unit; and when the object issensed from a forward direction of the main body, the controller 140 mayperform control to extract a central lower area of the image acquired bythe image acquisition unit.

In addition, the controller 140 may perform control to shift anextraction target area of an image acquired by the image acquisitionunit to correspond to a direction in which the sensed object is present,and then extract the extraction target area.

Meanwhile, input data for distinguishing an attribute of an object, anddata for training the DNN may be stored in the storage 150.

An original image acquired by the image acquisition unit 120, andextracted images of a predetermined area may be stored in the storage150.

In addition, in some implementations, weights and biases of the DNN maybe stored in the storage 150.

Alternatively, in some implementations, weights and biases of the DNNmay be stored in an embedded memory of the object recognition module144.

Meanwhile, whenever a part of an image acquired by the image acquisitionunit 120 is extracted, the object recognition module 144 may perform alearning process using the extracted part of the image as training data,or, after a predetermined number of images is extracted, the objectrecognition module 144 may perform a learning process.

That is, whenever an object is recognized, the object recognition module144 may add a recognition result to update a DNN architecture, such as aweight, or, when a predetermined number of training data is secured, theobject recognition module 144 may perform a learning process using thesecured data sets as training data so as to update an DNN architecture.

Alternatively, the mobile robot 100, 100 a, or 100 b transmits anoriginal image acquired by the image acquisition unit 120 or anextracted part of the image to a predetermined server through thecommunication unit 190, and receive machine learning-related data fromthe predetermined server.

In this case, the mobile robot 100, 100 a, or 100 b may update theobject recognition module 141 based on the machine learning-related datareceived from the predetermined server.

FIG. 8 is an exemplary schematic internal block diagram of a serveraccording to an embodiment of the present invention.

Referring to FIG. 8, a server 70 may include a communication unit 820, astorage 830, a learning module 840, and a processor 810.

The processor 810 may control overall operations of the server 70.

Meanwhile, the server 70 may be a server operated by a manufacturer of ahome appliance such as the mobile robot 100, 100 a, or 100 b, a serveroperated by a service provider, or a kind of Cloud server.

The communication unit 820 may receive a diversity of data, such asstate information, operation information, controlled information, etc.,from a mobile terminal, a home appliance such as the mobile robot 100,100 a, or 100 b, etc., a gate way, or the like.

In addition, the communication unit 820 may transmit data responsive tothe received diversity of data to a mobile terminal, a home appliancesuch as the mobile robot 100, 100 a, 100 b, etc., a gate way or thelike.

To this end, the communication unit 820 may include one or morecommunication modules such as an Internet module, a communicationmodule, etc.

The storage 830 may store received information and data necessary togenerate result information responsive to the received information.

In addition, the storage 830 may store data used for learning machine,result data, etc.

The learning module 840 may serve as a learning machine of a homeappliance such as the mobile robot 100, 100 a, or 100 b, etc.

The learning module 840 may include an artificial network, for example,a Deep Neural Network (DNN) such as a Convolutional Neural Network(CNN), a Recurrent Neural Network (RNN), a Deep Belief Network (DBN),etc., and may train the DNN.

As a learning scheme for the learning module 840, both unsupervisedlearning and supervised learning may be used.

Meanwhile, according to a setting, the controller 810 may performcontrol to update an artificial network architecture of a home appliancesuch as the mobile robot 100, 100 a, or 100 b, into a learned artificialneural network architecture.

FIGS. 9 to 12 are diagrams for explanation of deep learning.

Deep learning is a kind of machine learning, and goes deep inside datain multiple stages.

The deep learning may represent a set of machine learning algorithmsthat extract more critical data from a plurality of data sets on eachlayer than on the previous layer.

Deep learning architectures may include an ANN and may be, for example,composed of a DNN such as a Convolutional Neural Network (CNN_, aRecurrent Neural Network (RNN), a Deep Belief Network (DBN), etc.

Referring to FIG. 9, the ANN may include an input layer, a hidden layer,and an output layer. Each layer includes a plurality of nodes, and eachlayer is connected to a subsequent layer. Nodes between adjacent layersmay be connected to each other with having weights.

Referring to FIG. 10, a computer (machine) constructs a feature map bydiscovering a predetermined pattern from input data 1010. The computer(machine) may recognize an object by extracting a low level feature 1020through an intermediate level feature 1030 to an upper level feature1040, and output a corresponding result 1050.

The ANN may abstract a higher level feature on a further subsequentlayer.

Referring to FIGS. 9 and 10, each node may operate based on anactivation model, and an output value corresponding to an input valuemay be determined according to the activation model.

An output value of a random node, for example, the low level feature1020, may be input to a subsequent layer connected to the correspondingnode, for example, a node of the intermediate level feature 1030. Thenode of the subsequent layer, for example, the node of the intermediatelevel feature 1030, may receive values output from a plurality of nodesof the lower level feature 1020.

At this time, a value input to each node may be an output value of aprevious layer with a weight applied. A weight may refer to strength ofconnection between nodes.

In addition, a deep learning process may be considered a process fordetermining an appropriate weight.

Meanwhile, an output value of a random node, for example, theintermediate level feature 1030, may be input to a node of a subsequentlayer connected to the corresponding node, for example, a node of thehigher level feature 1040. The node of the subsequent layer, forexample, the node of the higher level feature 1040, may receive valuesoutput from a plurality of nodes of the intermediate level feature 1030.

The ANN may extract feature information corresponding to each levelusing a trained layer corresponding to each level. The ANN may performabstraction sequentially such that a predetermined object may berecognized using feature information of the highest layer.

For example, face recognition with deep learning is as follows: acomputer may classify bright pixels and dark pixels by degrees ofbrightness from an input image, distinguish simple forms such asboundaries and edges, and then distinguish more complex forms andobjects. Lastly, the computer may recognize a form that defines a humanface.

A deep learning architecture according to the present invention mayemploy a variety of well-known architectures. For example, the deeplearning architecture may be a CNN, RNN, DBN, etc.

The RNN is widely used in processing natural languages and efficient inprocessing time-series data, which changes over time, and thus, the RNNmay construct an artificial neural network architecture by piling uplayers each time.

The DBN is a deep learning architecture that is constructed by piling upmultiple layers of Restricted Boltzman Machine (RBM) which is a deeplearning technique. If a predetermined number of layers is constructedby repeating RBM learning, a DBN having the predetermined number oflayers may be constructed.

The CNN is an architecture widely used especially in object recognitionand the CNN will be described with reference to FIGS. 11 and 12.

The CNN is a model that imitates human brain functions based on anassumption that a human extracts basic features of an object, performscomplex calculation on the extracted basic features, and recognizes theobject based on a result of the calculation.

FIG. 11 is a diagram of a CNN architecture.

A CNN may also include an input layer, a hidden layer, and an outputlayer.

A predetermined image is input to the input layer.

Referring to FIG. 11, the hidden layer may be composed of a plurality oflayers and include a convolutional layer and a sub-sampling layer.

In the CNN, a pooling or non-linear activation function is used to addnon-linear features and various filters so as to basically extractfeatures of an image through convolution computation.

Convolution is mainly used in filter computation in image processingfields and used to implement a filter for extracting a feature from animage.

For example, as in FIG. 12, if convolution computation is repeatedlyperformed with respect to the entire image by moving a 3×3 window, anappropriate result may be obtained according to a weight of the window.

Referring to (a) of FIG. 12, if convolution computation is performedwith respect to a predetermined area 1210 using a 3×3 window, a result1201 is outcome.

Referring to (b) of FIG. 12, if computation is performed again withrespect to an area 1220 by moving the 3×3 window to the right, apredetermined result 1202 is outcome.

That is, as shown in (c) of FIG. 12, if computation is performed withrespect to an entire image by moving a predetermined window, a finalresult may be obtained.

A convolution layer may be used to perform convolution filtering forfiltering information extracted from a previous layer with a filter of apredetermined size (for example, a 3×3 window exemplified in FIG. 12).

The convolutional layer performs convolutional computation on inputimage data 1100 and 1102 with a convolutional filter, and generatesfeature maps 1101 and 1103 representing features of an input image 1100.

As a result of convolution filtering, filtered images in the same numberof filters included in the convolution layer may be generated. Theconvolution layer may be composed of nodes included in the filteredimages.

In addition, a sub-sampling layer paired with a convolution layer mayinclude feature maps in the same number as feature maps of the pairedconvolutional layer.

The sub-sampling layer reduces dimension of the feature maps 1101 and1103 through sampling or pooling.

The output layer recognizes the input image 1100 by combining variousfeatures represented in a feature map 1104.

An object recognition module of a mobile robot according to the presentinvention may employ the aforementioned various deep learningarchitectures. For example, a CNN widely used in recognizing an objectin an image may be employed, but aspects of the present invention arenot limited thereto.

Meanwhile, training of an ANN may be performed by adjusting a weight ofa connection line between nodes so that a desired output is output inresponse to a given input. In addition, the ANN may constantly updatethe weight due to the training. In addition, back propagation or thelike may be used to train the ANN.

FIGS. 13 and 14 are diagrams for explanation of object recognition ofthe object recognition module 144.

Referring to FIG. 13, the object recognition module 144 may classifyobjects into classes such as a fan, a home theater, a multi plug, a lampbase, a human hair, a bump, etc., and recognize the objects in adistinguishable manner.

In addition, the object recognition module 144 may classify the classessuch as a fan, a home theater, a multi plug, a lamp base, a human hair,etc. as a dangerous object super-class which is a higher conceptuallevel, and may recognize the object in a distinguishable manner.

In addition, the object recognition module 144 may classify an object,such as a bump and the like, which the mobile robot is able to travelforward, as a non-dangerous object super-class and recognize the objectin a distinguishable manner.

Referring to (a) of FIG. 14, the object recognition module 144 mayrecognize an input image and obtain a recognition result indicative of afan with a confidence of 0.95 and a recognition result indicative of ahome theater with a confidence of 0.7. In this case, the objectrecognition module 144 may output the fan, which is a recognition resulthaving a higher confidence of 0.7, as a recognition result with respectto the input image.

Meanwhile, a confidence may be normalized within a range from 0.0 to1.0.

Referring to (b) of FIG. 14, the object recognition module 144 mayrecognize an input image and obtain a recognition result indicative of afan with a confidence of 0.35 and a recognition result indicative of ahome theater with a confidence of 0.4.

For example, in the case where it is set to ignore a confidence of 0.6or less, the object recognition module 144 may not select any specificrecognition result because the two recognition results are lower than areference threshold, and the object recognition module 144 may determineunknown data.

Referring to (c) of FIG. 14, the object recognition module 144 mayrecognize an input image and obtain a recognition result indicative of afan with a confidence of 0.95 and a recognition result indicative of ahome theater with a confidence of 0.9.

For example, in the case where it is set to select a recognition resultwith a confidence of 0.9 or higher as a final recognition result, theobject recognition module 144 may not select any specific recognitionresult because the two recognition results are higher than a referencethreshold, and thus, the object recognition module 144 may determine anobject as a dangerous object which is a higher conceptual level.

Alternatively, in the case where it is set to confirm a higherrecognition result when a difference between confidences is 0.15 orhigher, the object may be determined as a dangerous object that is ahigher conceptual level.

Meanwhile, even in the case where an object is determined as a dangerousobject, the travel control module 141 may control the travel drive unit160 so as to move while detouring around the dangerous object.

FIG. 15 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

Referring to FIGS. 2 to 7 and 15, the mobile robot 100, 100 a, or 100 bmay perform cleaning while moving according to a command or a setting(S1510).

Meanwhile, the image acquisition unit 120 may acquire a plurality ofimages by continuously photographing the surroundings of the main body110 during traveling (S1520). The plurality of continuous imagesacquired by the image acquisition unit 120 may be stored in the storage150.

When an object is sensed by the sensor unit 170 during movement (S1530),the controller 140 may select an image acquired at a specific point intime earlier than an object sensing time of the sensor unit 170 fromamong the plurality of continuous images (S1540) based on a movingdirection and a moving speed of the main body 110.

The moving direction and the moving speed may be calculated by thetravel control module 141 or the like based on an output from anoperation detection sensor of the sensor unit 170.

Meanwhile, in the case where the mobile robot is set to travel at areference speed in normal travel, the moving speed is constant, andthus, it is possible to select an image acquired at a specific point intime by determining a moving direction of the main body 110.

Meanwhile, in the case where the moving direction corresponds to forwardtravel or rotational travel at an angle smaller than a predeterminedreference threshold (reference angle), the controller 140 may select animage acquired at a specific point in time earlier than the objectsensing time of the sensor unit 170 from the plurality of continuousimages.

In general, a sensing range of a sensor such as an ultrasonic sensor isgreater than a photographing range of a camera.

Accordingly, in the case where an image is acquired in response to asignal indicative of sensing of an object by the sensor unit 170, themobile robot is still traveling and thus the acquired image may notinclude features of the object. In addition, in the case where a sensingrange of the sensor unit 170 is short, an acquired image is more likelynot to include features of an object.

Accordingly, in the case of forward travel or approximately forwardtravel, the controller 140 may select an image acquired at a specificpoint in time earlier than the object sensing time of the sensor unit170 based on a moving direction and a moving speed.

In this case, if the moving speed is slower, the controller 140 mayselect an image acquired at a point in time further earlier than theobject sensing time of the sensor unit 170.

If the moving speed is faster, the mobile robot 100 travels a longerdistance after the object sensing time of the sensing time 170. In otherwords, when the image acquisition unit 120 acquires a plurality ofimages at a constant speed, the mobile robot travels a longer distanceduring a period of time between a point in time to capture a specificframe and a point in time to capture a subsequent frame.

Therefore, as a moving speed is faster the probability that an areaoccupied by an object in an image is large is higher when the image atthe point in time closer to an object sensing time of the sensor unit170 is selected.

In contrast, as a moving speed decreases, the image acquisition unit 120acquires more images while traveling the same distance, and thus, it maybe preferable to select an image captured at a point in time furtherearlier than an object sensing time of the sensor unit 170.

The object recognition module 144 may select the image acquired at thespecific point in time, and use the image as input data for recognitionof the object.

Meanwhile, the object recognition module 144 trained through the machinelearning exhibits a higher recognition rate if a learned object occupiesa higher proportion of input image data.

Thus, the controller 140 may perform control to crop part of theselected image acquired at the specific point in time in correspondenceto a direction in which an object sensed by the sensor unit 170 ispresent, and to extract the cropped part (S1550).

As a different part is extracted from an image acquired by the imageacquisition unit 120 depending on a direction in which an object sensedby the sensor unit 170 is present, and the extracted part of the imageis used as data for recognition, a recognition rate may be enhanced.

The object recognition module 144 may recognize an attribute of theobject included in an image of the part extracted from the selectedimage acquired at the specific point in time (S1560).

The object recognition module 144 may include an ANN having trained torecognize attributes such as types of objects through machine learning,and may recognize an attribute of an object included in an image of apart extracted from the selected image acquired at the specific point intime based on data that is pre-learned through machine learning (S1560).

For example, a CNN, which is one of deep learning architectures, may beembedded in the object recognition module 144, and a pre-trained CNN mayrecognize an attribute of an object included in input data and output aresult of the recognition.

Meanwhile, by performing the aforementioned object recognition processrepeatedly and achieving a final recognition result based on a pluralityof recognition results, it is possible to further enhance recognitionaccuracy.

For example, the controller 140 may store position information of asensed object and position information of the mobile robot in thestorage 150, and perform control such that an area of a predeterminedsize around the position of the sensed object is registered as an objectarea in a map.

Thereafter, whenever the mobile robot passes the registered object area,an attribute of the object may be recognized and a final attribute ofthe object may be determined based on the plurality of recognitionresults.

The controller 140 may recognize an attribute of an object sequentiallywith respect to images acquired by the image acquisition unit 120 in apredetermined object area, and determine a final attribute of the objectbased on such a plurality of recognition results of the sequentialrecognition. This embodiment will be described in more detail withreference to FIGS. 29 to 34.

Meanwhile, the travel control module 141 may control driving of thetravel drive unit 160 based on the recognized attribute of the object(S1570).

For example, if the recognized object is an object having a height thatthe mobile robot 100 is able to climb, the travel control module 141 mayperform control to travel while detouring around the object.

In addition, if the recognized object is an object, such as a low bump,having a height that the mobile robot 100 is able to climb, such as alow door threshold, the travel control module 141 may perform control tokeep traveling in a forward direction.

In addition, if an object is recognized as an object that can restrictthe mobile robot 100 during traveling even though the object is a lowheight object such as a fan base, a human hair, a multi plug, anelectric wire, etc., the travel control module 141 may perform controlto travel while detouring around the object.

FIG. 16 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

Referring to FIGS. 2 to 7 and 16, the mobile robot 100, 100 a, or 100 bmay move according to a command or a setting (S1610).

In the case where the sensor unit 170 includes an ultrasonic sensor, thesensor unit 170 may sense an object in response to sensing of areflected ultrasonic signal (S1620).

Meanwhile, the image acquisition unit 120 may acquire a plurality ofimages by constantly photographing the front area and surroundings ofthe mobile robot 100, 100 a, or 100 b.

The controller 140 may select an image acquired at a predeterminedprevious point in time from among a plurality of images acquired by theimage acquisition unit 120 in consideration of a moving direction and amoving speed, and determine, on the basis of data that is pre-learnedthrough machine learning, an attribute of an object sensed in theselected image acquired at the specific point in time.

In addition, the controller 140 may determine whether the sensed objecthas a height that the mobile robot 100 is able to climb (S1630).

If the recognized object is an object having a height that the mobilerobot is not able to climb (S1630), the controller 140 may performcontrol to travel while detouring the object by rotating by 90 degrees(S1655).

Meanwhile, if the recognized object is an object having a height thatthe mobile robot is able to climb (S1630), the controller 140 maydetermine attribute information of the sensed object (S1640). That is,the controller 140 may determine whether the recognized object is lesslikely to restrict the mobile robot and thus considered an object thatthe mobile robot 100 is able to climb.

If the recognized object is determined as an object that the mobilerobot is able to climb, the controller 140 may perform control to keepmoving forward (S1650).

Conventionally, whether a sensed object has a height that a mobile robotis able to climb is determined, and, when the sensed object is an objecthaving a low height, the mobile robot moves forward.

However, when an object such as an electric wire is present, a mobilerobot is sometimes caught and thus restricted by the electric wire.

In addition, the mobile robot may try to escape from the restrictedstate by performing a motion of shaking to the left/right, but this maylead to an accident, for example, peeling off of the electric wire.

However, the present invention recognizes object attribute informationwith learning machine and image information and determines a travelpattern depending on a recognized object attribute, thereby enhancingreliability.

FIGS. 17 to 25 are diagrams for explanation of a method for controllinga mobile robot according to an embodiment of the present invention.

FIG. 17 shows the case where an object 1700 is sensed from a forwarddirection of the mobile robot 100.

Referring to FIG. 17, the image acquisition unit 120 may acquire aplurality of images by continuously photographing while the mobile robot100 travels. The mobile robot 100 may acquire a first image 1711 at afirst position 1710, a second image 1721 at a second position 1720, anda third image 1731 at a third position 1730.

Referring to FIG. 18, a predetermined number of images may be stored inthe storage 150. In addition, if a preset number of images is acquired,an image acquired at the earliest point in time may be deleted and anewly acquired image may be stored.

The mobile robot 100 may start an image recognition processing inaccordance with a trigger signal responsive to sensing an ultrasonicsignal.

However, since an ultrasonic sensor has a short range, a feature of arecognition subject may not be found in the image 1731 acquired

Thus, the present invention may store continuous images in the storage150, determine whether a direction of travel is a forward direction, andperform an object recognition process not using the third image 1731acquired at the trigger event, but using the first image 1711.

Since the mobile robot 100 travels at a constant speed most of times, itis possible to perform an object recognition process simply bydetermining whether the mobile robot 100 is traveling forward, and byselecting an image acquired at a specific point in time, for example, aprevious image two frames earlier than an object sensing time of thesensor unit 170.

In addition, by taking into consideration even a sensing range of thesensor unit 170, performance, and a processing speed of the objectrecognition process, the controller 140 may determine how much earlier aprevious image is selected compared to the object sensing time of thesensor unit 170.

Meanwhile, in some embodiments, by using only a part of the imageacquired at the selected point in time, rather than the entire image, asinput data for object recognition, it is possible to enhance arecognition rate.

In this embodiment, it is possible to extract the center area, the leftarea, and the right area of an image based on a direction in which anobject is sensed, rather than cropping an area of a predetermined sizearound the center area of the image.

For example, when the object is sensed from a forward-right direction ofthe main body, a right lower area of the image acquired at the selectedpoint in time may be extracted; when the object is sensed from aforward-left direction of the main body, a left lower area of the imageacquired at the selected point in time may be extracted; and when theobject is sensed from a forward direction of the main body, a centrallower area of the image acquired at the selected point in time may beextracted.

Accordingly, objects as many as possible may be included in input datafor recognition. Since a machine recognizes what occupies the largestproportion of an image, it is possible to enhance a rate of recognitionof an object attribute.

Referring to (a) of FIG. 19, the sensor unit 170 according to anembodiment of the present invention may include a first sensor S1disposed on the front surface of the main body of the mobile robot, anda second sensor S2 and a third sensor S3 respectively spaced apart fromthe first sensor S1 in the left and right direction.

In this case, the first sensor S1 may operate as a transmitter, and thesecond sensor S2 and the third sensor S3 may operate as a receiver. Forexample, the first sensor S1 may transmit an ultrasonic signal, and thesecond sensor S2 and the third sensor S3 may receive a signal reflectedby an object. If the signal reflected by the object is received, it ispossible to determine a direction in which the object is present and adistance to the object with well-known recognition methods usingultrasound.

The drawing (a) of FIG. 19 shows an example in which an object X1 issensed at the center from a forward direction of the mobile robot 100.If a distance L1 from the sensed object X1 to the second sensor S2 and adistance L2 from the sensed object X1 to the third sensor S3 are equal(L1=L2), it is possible to determine that the object X1 is sensed at thecenter from the forward direction of the mobile robot.

In this case, as show in (b) of FIG. 19, it is possible to extract apredetermined area 1910 of size a2×b2 from the central lower side of anoriginal image 1900 of size a1×b1 acquired by the image acquisition unit120.

FIGS. 20 to 24 shows examples in which an object is recognized from aside.

FIG. 20 shows an example in which an object 2000 is sensed from aforward-right direction of the mobile robot 100.

Referring to FIGS. 20 and 21, the image acquisition unit 120 may acquirea plurality of images by keeping photographing while the mobile robot100 travels. The mobile robot 100 may acquire a first image 2110 at afirst position 2010, a second image 2120 at a second position 2020, anda third image 2130 at a third position 2030.

Referring to FIG. 22, a predetermined number of images may be stored inthe storage 150. In addition, if a preset number of images is acquired,an image acquired at the earliest point in time may be deleted and anewly acquired image may be stored.

The present invention may store continuous images in the storage 150,determine whether a direction of travel is a forward direction, andperform an object recognition process using not the third image 2130captured at the trigger event, but using the first image 2110, which iscaptured earlier than the third image 2130.

Meanwhile, it is possible to recognize an object by cropping andextracting a right area of the first image 2110, rather than using thefirst image 2110 intact.

The drawing (a) of FIG. 23 shows an example in which an object X1 issensed from a forward-right direction of the mobile robot 100, and thedrawing (a) of FIG. 24 shows an example in which the object X1 is sensedfrom a forward-left direction of the mobile robot 100.

In addition, the controller 140 may perform control to extract anextraction target area from an image acquired by the image acquisitionunit 120 by shifting the extraction target area in proportion to adifference between a distance L1 from the sensed object X1 to the secondsensor S2 and a distance L2 from the sensed object X1 and the thirdsensor S3.

Referring to (a) of FIG. 23, if the distance L1 from the sensed objectX1 to the second sensor S2 is greater than the distance L2 from thesensed object X1 to the third sensor S3 (L1>L2), it is possible todetermine that the object is sensed from the forward-right direction ofthe mobile robot 100.

In this case, as shown in (b) of FIG. 23, it is possible to extract apredetermined area 1920 of size a2×b2 from the right lower side of anoriginal image 1900 of size a1×b1 acquired by the image acquisition unit120.

In comparison between (b) of FIG. 19 and (b) of FIG. 23, it is foundthat a central point CP2 and a start point SP2 of the extraction targetarea 1920 of when the object X1 is sensed from the forward-rightdirection of the mobile robot 100 have been shifted by a predeterminedvalue d1 in a right direction from a central point CP1 and a start pointSP1 of the extraction target area 1910 of when the object X1 is sensedat the center from the forward direction of the mobile robot 100.

In this case, shifting by the predetermined value d1 may be proportionalto the difference (L1−L2) between the distance L1 from the sensed objectX1 to the second sensor S2 and the distance L2 from the sensed object X1to the third sensor S3.

Referring to (a) of FIG. 24, if the distance L1 from the sensed objectX1 to the second sensor S2 is smaller than the distance L2 from thesensed object X1 to the third sensor S3 (L1<L2), it is possible todetermine that the object X1 is sensed from a forward-left direction ofthe mobile robot 100.

In this case, as shown in (b) of FIG. 24, it is possible to extract apredetermined area 1930 of size a2×b2 from the left lower side of anoriginal image 1900 of size a1×b1 acquired by the image acquisition unit120.

In comparison between (b) of FIG. 19 and (b) of FIG. 24, it is foundthat a central point CP3 and a start point SP3 of the extraction targetarea 1930 of when the object X1 is sensed from the forward-leftdirection of the mobile robot 100 have been shifted by a predeterminedvalue d2 in the left direction from a central point CP1 and a startpoint SP1 of the extraction target area 1910 of when the object X1 issensed at the center from the forward direction of the mobile robot 100.

In this case, shifting by the predetermined value d2 may be proportionalto the difference (L2−L1) between the distance L1 from the sensed objectX1 to the second sensor S2 and the distance L2 from the sensed object X1to the third sensor S3.

Meanwhile, according to an embodiment of the present invention, thismethod may be applied not just to forward travel but also to rotationaltravel.

For example, if the rotational travel is rotation by an angle smallerthan a predetermined reference threshold, it is possible to select animage acquired at a specific point in time earlier than an objectsensing time of the sensor unit 170 from among the plurality ofcontinuous images.

Referring to FIG. 25, the mobile robot 100 may move while rotating froma first position 2510 through the second position 2520 to a thirdposition 2530. In this case, if the mobile robot 100 rotates by an anglesmaller than a predetermined reference threshold, an object recognitionprocess may be performed in the same manner as it does when the mobilerobot 100 travels in a forward direction.

For example, if the mobile robot 100 rotates within a field of view 2501from the first position 2510, the mobile robot 100 may perform an objectrecognition process in the same manner as when traveling in a forwarddirection.

The field of view 2501 may refer to a range in which the imageacquisition unit 120 is allowed to acquire images.

Meanwhile, as the mobile robot according to an embodiment of the presentinvention performs a learning process with training data which is theselected image acquired at the specific point in time or an image of anextracted part of the selected image acquired at the specific point intime, the mobile robot may constantly update architectures of an ANN anda DNN.

Alternatively, the mobile robot may transmit, to a predetermined server,the selected image acquired at the specific point in time or the imageof the extracted part of the selected image acquired at the specificpoint in time, and receive machine learning-related data from thepredetermined server. Thereafter, the mobile robot may update the objectrecognition module 144 based on the machine learning-related datareceived from the predetermined server.

FIG. 26 is a diagram for explanation of an operation method of a mobilerobot and a server according to an embodiment of the present invention,and FIG. 27 is a flowchart of an operation method of a mobile robot anda server according to an embodiment of the present invention.

Referring to FIGS. 26 and 27, a DNN architecture 144 a, such as a CNN,may be embedded in the object recognition module 144 of the mobile robot100.

A pre-learned DNN architecture 144 a may receive input data forrecognition (S2710), recognize an attribute of an object included in theinput data (S2720), and output a result of the recognition (S2730).

Unknown data unrecognizable by the DNN architecture 144 a may be storedin the storage 150 or a storage space 144 b within the objectrecognition module 144 (S2740).

Meanwhile, the unknown unrecognizable by the object recognition module144 may be transmitted to the server 70 through the communication unit190 (S2741). In addition, data recognized by the object recognitionmodule 144 may be transmitted to the server 70 as well.

The server 70 may generate configuration of learned weights, and theserver 70 may learn the DNN architecture with training data.

The server 70 may train the DNN based on the received data (S2742), andupdate the DNN by transmitting updated DNN architecture data to themobile robot 100 (S2743).

FIG. 28 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

Referring to FIG. 28, firstly, the mobile robot 100, 100 a, or 100 b mayperform cleaning while traveling according to a command or a setting(S2810).

Meanwhile, the image acquisition unit 120 may acquire a plurality ofimages by continuously photographing the surroundings of the main body110 during traveling (S2820). The plurality of continuous imagesacquired by the image acquisition unit 120 may be stored in the storage150.

If an object is sensed by the sensor unit 170 during traveling (S2830),the controller 140 may select an image acquired at a specific point intime earlier than an object sensing time of the sensor unit 170 fromamong the plurality of continuous images based on a moving direction anda moving speed of the main body 110 (S2840).

The moving direction and the moving speed may be calculated by thetravel control module 141 or the like based on an output of an operationdetection sensor of the sensing unit 170.

Meanwhile, in the case where it is set to travel at a reference speed innormal travel, the moving speed is constant, and thus, it is possible toselect an image acquired at a specific point in time by determining amoving direction of the main body 100.

Meanwhile, in the case where the moving direction corresponds to forwardtravel or rotational travel at an angle smaller than a predeterminedreference threshold (reference angle), the controller 140 may select animage acquired at a specific point in time earlier than the objectsensing time of the sensor unit 170 from the plurality of continuousimages.

In the case where an image is acquired in response to a signalindicative of sensing of an object by the sensor unit 170, the mobilerobot is still traveling and thus the acquired image may not include afeature of the object. In addition, in the case where a sensing range ofthe sensor unit 170 is short, an acquired image is more likely not toinclude a feature of an object.

Accordingly, in the case of forward travel or approximately forwardtravel, the controller 140 may select an image acquired at a specificpoint in time earlier than the object sensing time of the sensor unit170 based on a moving direction and a moving speed.

In this case, if the moving speed is slower, the controller 140 mayselect an image acquired at a point in time further earlier than theobject sensing time of the sensor unit 170.

If the moving speed is faster, the mobile robot 100 travels a longerdistance after the object sensing time of the sensing time 170. In otherwords, when the image acquisition unit 120 acquires a plurality ofimages at a constant speed, the mobile robot travels a longer distanceduring a period of time between a point in time to capture a specificframe and a point in time to capture a subsequent frame.

Therefore, as a moving speed increases, the probability that the areaoccupied by the object in the image is high is high in selecting theimage at the point in time when the sensor unit 170 detects the object.

In contrast, as a moving speed decreases, the image acquisition unit 120acquires more images while traveling the same distance, and thus, it maybe preferable to select an image captured at a point in time furtherearlier than an object sensing time of the sensor unit 170.

The object recognition module 144 may select the image acquired at thespecific point in time, and use the image as input data for recognitionof the object.

Based on data pre-learned through machine learning, the objectrecognition module 144 may recognize an attribute of an object includedin the selected image acquired at the specific point in time (S2850).

The object recognition module 144 may include an ANN trained torecognize attributes such as types of objects through machine learning,and recognize an attribute of an object based on pre-learned data(S2850).

For example, a CNN, which is one of deep learning architectures, may beembedded in the object recognition module 144, and a pre-trained CNN mayrecognize an attribute of an object included in input data and output aresult of the recognition.

Meanwhile, the travel control module 141 may control the travel driveunit 160 based on the recognized attribute of the object (S2860).

For example, if the recognized object is an object having a height thatthe mobile robot 100 is not able to climb, the travel control module 141may perform control to travel while detouring around the object.

In addition, if the recognized object is an object, such as a low bump,having a height that the mobile robot 100 is able to climb, such as alow door threshold, the travel control module 141 may perform control tokeep traveling in a forward direction.

In addition, if an object is recognized as an object that can restrictthe mobile robot 100 during traveling even though the object is a lowheight object such as a fan base, a human hair, a multi plug, anelectric wire, etc., the travel control module 141 may perform controlto travel while detouring around the object.

FIG. 29 is a flowchart of a method for controlling a mobile robotaccording to an embodiment of the present invention.

Referring to FIG. 29, the mobile robot 100, 100 a, or 100 b may performcleaning while moving according to a command or a setting (S2910).

The sensor unit 170 may include an object detection sensor 131 and sensean object within a predetermined range in a forward direction of themoving direction. (S2920).

If an object is sensed by the sensor unit 170 (S2920) during movement,position information of the sensed object and position information ofthe mobile robot 100, 100 a, or 100 b may be stored in the storage 150(S2930).

In addition, an area having a predetermined size around the position ofthe sensed object may be registered as an object area in a map stored inthe storage 150 to store and manage the area (S2930).

The position information of the mobile robot may be calculated fromodometry information, and the odometry information may be constructedbased on data sensed by the aforementioned operation sensor of thesensor unit 170.

Meanwhile, the image acquisition unit 120 may acquire a plurality ofimages in the object area during the movement (S2940).

The image acquisition unit 120 may register the object area, and acquirea plurality of images by continuously photographing images in the objectarea before the mobile robot 100, 100 a, or 100 b departs out of theobject area.

In addition, the mobile robot 100, 100 a, or 100 b may acquire an imageof the entire object area while performing avoidance driving withrespect to the sensed object.

Meanwhile, photographing an object area while traveling in the objectarea with a series of operations in order to explore the entire objectarea may bring significant change in a travel path of the mobile robot100, 100 a, or 100 b.

Thus, it is preferable that the mobile robot 100, 100 a, or 100 bacquires images about an object by photographing the object areawhenever passing the object area while traveling in normal travel.

For example, suppose that the mobile robot 100, 100 a, or 100 b travelsin a basic zigzag pattern by moving in a forward direction, changing themoving direction, and then moving a distance in a direction opposite tothe initial moving direction until sensing an object: in this case, themobile robot 100, 100 a, or 100 b may acquire a plurality of images in amanner of acquiring a predetermined number of images by photographingwhile passing an object area when first traveling forward, and in amanner of acquiring a predetermined number of images by photographingwhile passing the object area again when returning back to the initialposition.

Meanwhile, the object recognition module 144 may recognize an attributeof an object, based on data which is pre-learned through learningmachine, with respect to images acquired by the image acquisition unit120 during movement in the object area (S2950).

The object recognition module 144 may include an ANN having trained torecognize attributes such as types of objects through machine learning,and may recognize an attribute of an object based on pre-learned data(S2950).

For example, a CNN, which is one of deep learning architectures, may beembedded in the object recognition module 144, and a pretrained CNN mayrecognize an attribute of an object included in input data and output aresult of the recognition.

After performing an object recognition process with respect to one imageacquired in the object area and then performing an object recognitionprocess with respect to another image, the object recognition module 144may select and store a more accurate recognition result from tworecognition results regarding the two images.

Meanwhile, the object recognition module 144 may compare confidencesincluded in the two recognition results to select a more accuraterecognition result. A result of the last recognition process may be thecurrent object recognition result, and, when a recognition process isperformed with respect to a subsequent image, the current objectrecognition result may become a previous object recognition result and aresult of the recognition process with respect to the subsequent imagemay become the current object recognition result.

Meanwhile, when recognizing an attribute of an object sequentially withrespect to images acquired by the image acquisition unit 120 (S2950),the object recognition module 144 may repeatedly compare the currentobject recognition result and a previous object recognition result.

Whenever the mobile robot 100, 100 a, or 100 b passes the object area innormal travel, an image may be acquired by photographing the objectarea.

The object recognition module 144 may perform an image recognitionprocess with respect to acquired images in sequence, compare a result ofthe recognition with a previous object recognition result, and select arecognition result with a higher confidence.

Preferably, the object recognition module 144 may compare the currentobject recognition result and a previous object recognition result, and,if the current object recognition result and the previous objectrecognition result are identical, the object recognition module 144 maymaintain the current object recognition result and increase a confidenceby applying a predetermined weight.

That is, if the current object recognition result and the previousobject recognition result are identical, it is not necessary to changethe object recognition result, and therefore, it is possible to increasea confidence for the recognition result by applying the predeterminedweight, while maintaining the object recognition result.

Thus, whenever an identical recognition result is outcome, the objectrecognition module 144 may perform control such that a confidence isincreased by applying the predetermined weight and stored.

In addition, if the current object recognition result and the previousobject recognition result are not identical, the object recognitionmodule 144 may register a recognition result having a higher confidenceout of the current object recognition result and the previous objectrecognition result as the new current object recognition result.

That is, the object recognition module 144 may compare the tworecognition results, leave a recognition result with a higherconfidence, and compare the recognition result with the higherconfidence with a subsequent recognition result.

Meanwhile, the object recognition module 144 may determine a finalattribute of the object based on the plurality of recognition results ofthe sequential recognition (S2960).

While recognizing an object sequentially, the object recognition module144 may determine a final attribute of an object upon satisfaction of apredetermined condition. That is, the object recognition module 144 mayoutput, as a final recognition result, a current recognition result ofwhen the predetermined condition is satisfied.

After the predetermined condition is satisfied, an object recognitionprocess is no longer performed, and thus, a current recognition resultof when the predetermined condition is satisfied may be the last currentrecognition result.

For example, the last current recognition result at a time when objectrecognition is completed with respect to the entire object area, thatis, when recognition coverage of the object area reaches 100%, may bedetermined as a final attribute of the object.

In some embodiments, it may be set to finish the object recognitionprocess when the recognition coverages reaches a predetermined thresholdsuch as 90% and 95%.

Alternatively, the object recognition module 144 may compare the currentobject recognition result and a previous object recognition result, and,if the current object recognition result and the previous objectrecognition result are identical, the object recognition module 144 maymaintain the current object recognition result and change a confidenceinto an average between a confidence for the current object recognitionresult and a confidence for the previous object recognition result.

The present embodiment is an embodiment in which when the current objectrecognition result and the previous object recognition result areidentical, a confidence for the identical recognition results isdetermined as an average between the existing confidences of the tworecognition results.

Even in this embodiment, if the current object recognition result andthe previous object recognition result are not identical, the objectrecognition module 144 may perform control to register a recognitionresult with a higher confidence out of the current object recognitionresult and the previous object recognition as the new current objectrecognition result.

Meanwhile, the travel control module 141 may control driving of thetravel drive unit 160 based on a final attribute of the recognizedobject (S2980).

For example, if the recognized object is an object having a height thatthe mobile robot 100 is not able climb, the travel control module 141may perform control to travel while detouring around the object.

In addition, if the recognized object is an object, such as a low bump,having a height that the mobile robot 100 is able to climb, the travelcontrol module 141 may perform control to keep traveling in a forwarddirection.

In addition, if an object is recognized as an object that can restrictthe mobile robot 100 during traveling even though the object is a lowheight object such as a fan base, a human hair, a multi plug, anelectric wire, etc., the travel control module 141 may perform controlto travel while detouring around the object.

Meanwhile, depending on the determined final attribute of the object,the object recognition module 144 may register and manage thecorresponding object as a dangerous object or a non-dangerous object inthe map (S2970).

For example, if the recognized object is an object having a height thatthe mobile robot 100 is not able to climb or if the recognized object isan object that can restrict the mobile robot 100 during traveling eventhough the object is a low height object such as a fan base, a humanhair, a multi plug, an electric wire, etc., the object recognitionmodule 144 may register the recognized object as a dangerous object inthe map.

In addition, an object such as a bump that the mobile robot is able toclimb when traveling in a forward direction may be registered as anon-dangerous object in the map.

Thereafter, based on information on a dangerous object and anon-dangerous object registered in the map, the mobile robot 100 maytravel while detouring around the dangerous object.

FIGS. 30 to 34 are diagrams for explanation of a method for controllinga mobile robot according to an embodiment of the present invention.

Referring to FIG. 30, the mobile robot 100 may sense an object 3001while traveling. When the mobile robot 100 senses the object 3001, anobject area may be set and registered to include the surroundings of therecognized object.

Referring to FIG. 30, the object recognition module 144 may performcontrol to register, in a map to be stored in the storage 150, an objectarea 3010 having a predetermined size a×b around an object 3001 in anentire area 3000 is registered.

For example, the object area 3010 may be set to be in a rectangularshape of 1 m×1 m having each front/rear edge of 50 cm and eachleft/right edge of 50 cm around the object 3001.

Meanwhile, in addition to the registration of the object area 3010, theobject recognition module 144 may perform control to store positioninformation of the mobile robot 100 of when the object 3001 isrecognized. Meanwhile, the position information of the mobile robot 100may be information based on odometry information.

For example, the object recognition module 144 may perform control tostore position information of an object and position information of themobile robot in the following save format.

Save Format: (Op, Rp),

Op(X,Y)=Position of Object

Rp(X,Y)=Position of Robot

Meanwhile, the image acquisition unit 120 may acquire a plurality ofimages in the object area 3010 during traveling.

The image acquisition unit 120 may acquire a plurality of images bycontinuously photographing in the object area 3010 before the mobilerobot 100 departs out of the object area 3010.

FIG. 31 shows that a predetermined area of the object area 3010 isrecognized by two times of photographing. Circles 3110 and 3120 in FIG.31 shows recognition coverages, and sizes and shapes thereof may differdepending on configuration and performance of the image acquisition unit120.

Meanwhile, the mobile robot 100 may photograph the object 3001 at adifferent position in the object area 3010.

According to the present invention, by acquiring images of the object3001 at different positions during movement in the object area 3010 andperforming an object recognition process with respect to each acquiredimage, it is possible to enhance object recognition accuracy.

Meanwhile, the object recognition module 144 may manage the inner areaof the object area 3010 by dividing the inner area in a grid patternwith constant vertical and horizontal spacing.

FIG. 32 shows an example of an object area 3200 in 4×4 grid pattern.

As an image of the inside of the object area 3200 is photographed, theobject recognition module 144 may distinguishably store an area in whichan object is recognized.

For example, as shown in (a) of FIG. 32, it is possible to distinguish arecognized area and a non-recognized area in the entire object area 3200by changing a value of a predetermined area 3210 which has beenrecognized, and it is possible to determine the extent to which arecognition process has been proceeded in the entire object area 3200.

Meanwhile, depending on traveling and object recognition of the mobilerobot 100, the number of recognized areas 3210, 3220, 3230, and 3240 mayincrease, as shown in (b) of FIG. 32.

In addition, as shown in (c) of FIG. 32, when the recognition coveragereaches 100%, that is, when the object recognition process is completedin the entire object area 3200, the object recognition process may beterminated.

Alternatively, according to a setting, when the recognition coveragereaches 90% or 95%, the object recognition process may be terminated.

FIGS. 33 and 34 show part of a sequential object recognition processwith respect to the same object according to an embodiment of thepresent invention.

Referring to FIGS. 33 and 34, firstly, the object recognition module 144may recognize an image acquired at a first position Rp1 and recognize anobject as a fan. For example, a current recognition result indicative ofrecognition of a fan may have a confidence of 0.8.

Meanwhile, the aforementioned recognition result is the firstrecognition result and no other comparative subject exists, andtherefore, a registration result corresponding to the currentrecognition result may be a fan.

Meanwhile, secondly, the object recognition module 144 may obtain arecognition result indicative of a fan with a confidence of 0.7, basedon an image acquired at a second position Rp2 of the mobile robot.

Since the first recognition result and the second recognition result areidentical, the object recognition module 144 may maintain a fan as theregistration result corresponding to the current recognition result.

In this case, one of confidences for the two recognition results may beused as a confidence for the current recognition result.

Alternatively, as shown in FIG. 33, 0.7 which is an average ofconfidences of the first recognition result and the second recognitionresult may be used as a confidence for the current recognition result.

More preferably, since the first recognition result and the secondrecognition result identically indicate a fan, a confidence of 0.835which has been increased with a predetermined weight applied, as shownin FIG. 25, may be used as a confidence for the current recognitionresult.

Referring to FIGS. 33 and 34, based on an image acquired at a thirdposition Rp3 by the mobile robot, the object recognition module 144 mayobtain a recognition result indicative of a lamp base with a confidenceof 0.7.

Referring to FIGS. 33 and 34, the object recognition module 144 maycompare the third recognition result with a previous recognition result.

As the confidence of 0.7 for the third recognition result is lower thana confidence of 0.75 or 0.835 for a previous recognition result, theobject recognition module 144 may select a fan having a higherconfidence as a registration result corresponding to the currentrecognition result.

According to at least one of the embodiments of the present invention,it is possible to acquire image data that helps increase accuracy ofrecognizing an attribute of an object.

According to at least one of the embodiments of the present invention,since a mobile robot is capable of determining an attribute of an objectand adjusting a traveling pattern according to the attribute of theobject, it is possible to perform highly reliable object recognition andavoidance.

In addition, according to at least one of the embodiments of the presentinvention, it is possible to provide a mobile robot and a method forcontrolling the mobile robot, the mobile robot which is capable ofmoving forward, moving backward, stopping, detouring, and the likeaccording to a recognition result on an object, thereby enhancing userconvenience and improving operational efficiency and cleaningefficiency.

In addition, according to at least one of the embodiments of the presentinvention, it is possible to provide a mobile robot and a method forcontrolling the mobile robot, the mobile robot which is capable ofaccurately recognizing an attribute of an object based on learningmachine.

In addition, according to at least one of the embodiments of the presentinvention, it is possible for a mobile robot to perform machine learningefficiently and extract data for recognition

As can be seen from the foregoing, the mobile robot according to thepresent invention is not limited to the configuration and method of theembodiments described above, but all or some of the embodiments may beconfigured to be selectively combined such that various modifications ofthe embodiments can be implemented.

Meanwhile, the mobile robot according to the embodiments of the presentinvention may be implemented in a recording medium, which may be read bya processor provided in a network device, as a code that can be read bythe processor. The recording medium that can be read by the processorincludes all kinds of recording media in which data that can be read bythe processor are stored. Examples of the recording medium include aROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical datamemory, etc., and may be implemented in a shape of carrier wave such astransmission through Internet. Also, the recording medium that can beread by the processor may be distributed in a computer system connectedthereto through the network, whereby codes that can be read by theprocessor may be stored and implemented in a distributive mode.

While the disclosed embodiment has been particularly shown and describedwith reference to exemplary aspects thereof, it will be understood thatvarious changes in form and details may be made therein withoutdeparting from the spirit and scope of the following claims, and Itshould be understood that many variations and modifications of the basicinventive concept described herein will still fall within the spirit andscope of the present invention as defined in the appended claims andtheir equivalents.

What is claimed is:
 1. A mobile robot comprising: a travel drive unitconfigured to move a main body; an image acquisition unit configured toacquire a plurality of images by continuously photographing surroundingsof the main body; a storage configured to store the plurality ofcontinuous images acquired by the image acquisition unit; a sensor unithaving one or more sensors configured to sense an object during themovement of the main body; and a controller is configured to, when thesensor unit senses the object, select, based on a moving direction and amoving speed of the main body, an image acquired at a specific point intime earlier than an object sensing time of the sensor unit from amongthe plurality of continuous images, and recognize an attribute of theobject included in the selected image acquired at the specific point intime.
 2. The mobile robot of claim 1, wherein the controller furthercomprises a travel control module configured to control driving of thetravel control unit based on the recognized attribute of the object. 3.The mobile robot of claim 1, wherein the controller is furtherconfigured to, when the moving direction corresponds to forward travelor rotational travel at an angle smaller than a predetermined referencethreshold, select the image acquired at the specific point in timeearlier than the object sensing time of the sensor unit from among theplurality of continuous images.
 4. The mobile robot of claim 1, whereinthe controller is further configured to select an image acquired at apoint in time further earlier than the object sensing time of the sensorunit as the moving speed is slower.
 5. The mobile robot of claim 1,wherein the object recognition module is further configured torecognize, based on data pre-learned through machine learning, anattribute of an object included in the selected image acquired at thespecific point in time.
 6. The mobile robot of claim 1, furthercomprising a communication unit configured to transmit the selectedimage acquired at the specific point in time to a predetermined server,and receive machine learning-related data from the predetermined data.7. The mobile robot of claim 6, wherein the object recognition module isfurther configured to be updated based on the machine learning-relateddata received from the predetermined server.
 8. The mobile robot ofclaim 1, wherein the controller further comprises an image processingmodule configured to extract a part of the selected image acquired atthe specific point in time in correspondence to a direction of theobject sensed by the sensor is present.
 9. The mobile robot of claim 8,wherein the image processing module is further configured to: when theobject is sensed from a forward-right direction of the main body,extract a right lower area of the selected image acquired at thespecific point in time; when the object is sensed from a forward-leftdirection of the main body, extract a left lower area of the selectedimage acquired at the specific point in time; and when the object issensed from a forward direction of the main body, extract a centrallower area of the selected image acquired at the specific point in time.10. The mobile robot of claim 1, wherein the controller is furtherconfigured to: store position information of the sensed object andposition information of the mobile robot in the storage; register anarea of a predetermined size around a position of the sensed object asan object area in a map; recognize an attribute of the objectsequentially with respect to images acquired by the image acquisitionunit in a predetermined object area; and determine a final attribute ofthe object based on a plurality of recognition results of the sequentialrecognition.
 11. A method for controlling a mobile robot, comprising:acquiring a plurality of images by continuously photographingsurroundings of a main body during traveling; storing the plurality ofcontinuous images acquired by the image acquisition unit; sensing anobject by the sensor unit; when the sensing unit senses the object,select, based on a moving direction and a moving speed of the main body,an image acquired at a specific point in time earlier than an objectsensing time of the sensor unit from among the plurality of continuousimages; recognizing an attribute of the object included in the selectedimage acquired at the specific point in time; and controlling driving ofthe travel control unit based on the recognized attribute of the object.12. The method of claim 11, wherein the selecting of the image acquiredat the specific point in time comprises, when the moving directioncorresponds to forward travel or rotational travel at an angle smallerthan a predetermined reference threshold, selecting the image acquiredat the specific point in time earlier than the object sensing time ofthe sensor unit from among the plurality of continuous images.
 13. Themethod of claim 11, wherein the selecting of the image acquired at thespecific point in time comprises selecting an image acquired at a pointin time further earlier than the object sensing time of the sensor unitas the moving speed is slower.
 14. The method of claim 11, wherein therecognizing of the attribute of the object comprises recognizing, basedon data pre-learned through machine learning, attribute of an objectincluded in the selected image acquired at the specific point in time.15. The method of claim 11, further comprises: transmitting the selectedimage acquired at the specific point in time to a predetermined server;and receiving machine learning-related data from the predetermined data.16. The method of claim 15, further comprising updating the objectrecognition module based on the machine learning-related data receivedfrom the predetermined server.
 17. The method of claim 11, furthercomprising extracting a part of the selected image acquired at thespecific point in time in correspondence to a direction of the objectsensed by the sensor unit is present.
 18. The method of claim 17,wherein the extracting of the part of the selected image acquired at thespecific point in time comprises: when the object is sensed from aforward-right direction of the main body, extract a right lower area ofthe selected image acquired at the specific point in time; when theobject is sensed from a forward-left direction of the main body, extracta left lower area of the selected image acquired at the specific pointin time; and when the object is sensed from a forward direction of themain body, extract a central lower area of the selected image acquiredat the specific point in time.
 19. The method of claim 11, wherein thecontrolling of the driving comprises performing control to perform anavoidance operation when the sensed object is not an object that themobile robot is able to climb.
 20. The method of claim 11, furthercomprising: when the sensor unit senses an object, storing positioninformation of the sensed object and position information of the mobilerobot in the storage and registering an area having a predetermined sizearound a position of the sensed object as an object area in a map;recognizing an attribute of the object sequentially with respect toimages acquired by the image acquisition unit during movement in theobject area; and determining a final attribute of the object based on aplurality of recognition results of the sequential recognition.